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Abstract 

In this paper we relate dynamics associated to zero-sum games (Fictitious play) to Hamil- 
tonian dynamics. It turns out that the Hamiltonian dynamics which is induced from fictitious 
play, has properties which are rather different from those found in more classically defined 
Hamiltonian dynamics. Although the vectorfield is piecewise constant (and so the flow (j)t 
piecewise a translation), the dynamics is rather rich. For example, there exists a Hamilton 
H so that for each t>Q the level set H~^{t) is homeomorphic to S'^ (the level sets consist of 
pieces of hyperplanes in R*) and with the following property. There exists a periodic orbit F 
of the Hamiltonian flow in H~^{1) so that the first return map _F to a section Z C -ff~^(l) 
transversal to F at i G F acts as a random-walk: there exist a nested sequence of annuli An 
in Z (around x so that UA„ U {a::} is a neighbourhood of x in Z) shrinking geometrically to 
X so that for each sequence n{i) > with \n{i + 1) — < 1 there exists a point z £ Z so 
that F'{z) e for aU i > 0. 

1 Introduction and statement of results 

In this paper we will introduce a rather unusual class of Hamiltonian systems. These are moti- 
vated by dynamics, usually referred to as Fictitious Play, associated to zero-sum games. These 
Hamiltonian systems differ from those that are considered traditionally, in that their orbits con- 
sist of piecewise straight lines and first return maps to certain planes are piecewise isometries. 
Specifically, the Hamiltonian systems we consider are continuous and piecewise affine, and defined 
by the Hamiltonian H : E x S ^ M, 

H{p, q) — ma.xp' Ai q ~ minp' A4 q. (1-1) 

pes ges 

Here A4 is a n x n matrix, p,q E E, the set of probability vectors in M" and p' stands for 
the transpose of p. The function H is continuous and piecewise affine {H is affine outside 
some finite union of hyperplanes) . Under some very general assumption, the level sets of H are 
homeomorphic to S^"~^ and made up of pieces of some hyperplanes, see Theorem 

In this paper we concentrate on matrices for which there exist p,q G T, so that p' Ai and A4 q 
are multiplies of w = (1 1 ... 1) resp. its transpose v' and so that all components of p and q are 
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strictly positive. In this case the pair {p,q) is called a completely mixed Nash equilibrium, 
due to its game-theoretic interpretation, see Section [2] 

In some of the theorems we shall need to make a transversal! ty assumption. Since in Theo- 
rem |1.5| the setting is slightly more general, let us assume that Ai is a m x n matrix with m 
and n not necessarily equal. The transversality assumption we will make is that 

each minor of A4 and of A4' is non-zero (1-2) 

where M ' is some (any) (to— 1) x n (resp. nx {n— 1)) matrix obtained by subtracting some row 
of A4 from all other rows of Ai (resp some column from all other columns). These conditions 
are open and dense (and of full Lcbesgue measure). Note that if this condition holds, then the 
Nash equilibrium is unique, see Lemmas |2 . 1 1 and |2 .2| 



1.1 A class of new Hamiltonian systems 

The first aim of this paper is to show that this class of Hamiltonian flows is rather interesting, and 
secondly to show that these flows are indeed related to flctitious play. Theorem |1.1| shows that 
when rather weak conditions are satisfied on M , the Hamiltonian flow is continuous. Then we 



state Theorems 1.2 and 1.3 which cite results from van Strien and Sparrow |2009j which show that 
such Hamiltonian systems have rather different dynamics from classical Hamiltonian systems. 
Theorem |1.4| shows that these Hamiltonian system correspond to Fictitious play associated to 
the two-player nxn zero-sum game with matrix Ai , and Theorem |1 .5| deal with the more general 
two-player mxn zero-sum games, showing that the Nash equilibria are unique and that the flow 



associated to Fictitious play is continuous, provided (1.2) holds 



Theorem 1.1 {H defines a continuous piecewise translation flow). Assume that Ai is a n x n 



matrix satisfies the transversality assumption {1.2) and let H: S x E — > M 6e as m (1.1). Then 
the following properties hold. 

• Each level set H^^(c) (taking c> 0) is topologically a sphere S'^"^^ (made up of pieces of 
hyperplanes m S x Y.); moreover, H^^{0) ~ {p,q}. 

• The Hamiltonian flow 

dp _ OH 

dt dq ' ON 

dq __dH_ ^^■'^> 

dt dp 

associated to H : Y, x Y, —t M. defines a continuous flow without stationary points on each 
level set H~^{c), c > 0. 

• The flow is piecewise a translation flow. 

• First return maps between certain hyperplanes are piecewise affine maps. 

3H dH 

Note that to compute the derivatives — — and — — one needs to take into account that p, q 

dq dp 

are constrained to lie in S. 
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The Hamiltonian systems which appear in this way, can have rather remarkable behaviour. 
We illustrate this by considering an example in the next theorem. This example is part of a 
family of systems studied in van Strien and Sparrow |2009j in the context of dynamical systems 
associated to game theory. Here we only will consider the situation which corresponds to the 
zero-sum case. Consider 

/ I f3\ 

P 1 (1.4) 

V /3 1 ; 

where P ^ —1,1 and let Ai be some matrix close to A. Note that A satisfies the above 
transversality assumption ( 1.2 1 (here we use that det(yl) = 1 + f3^) and that its Nash equilibrium 



(p, q) £ E X S corresponds to a vector with all components equal to 1/3. Then 

Theorem 1.2 (The Hamiltonian flow acts 'like a random walk': an example in R'^). Let H he the 
Hamiltonian assoc iated to a matrix Ai as in (1-1^ where M is sufficiently close to the matrix 



A defined in (1-4-) where (3 is taken as (3 — a := {^/b — l)/2 ~ 0.618. Then the set H~^{1) is 
homeomorphic to and the Hamiltonian flow associated to H has a periodic orbit T (a hexagon 
in H~^{1) w S^) with the following properties. If one takes the first return map F to a section 
Z transversal to T (through some point x (zT), then for each fc G N 

• there exists a sequence of periodic points Xn (z Z of exactly period k of the first return to Z 
accumulating to x; 

• the first return map F to Z has infinite topological entropy. 

• The dynamics acts as a random-walk. More precisely, there exist annuli An in Z (around 
T n Z so that UAn U {x} is a neighbourhood of x in Z) shrinking geometrically to T (1 Z so 
that for each sequence n{i) > with \n{i + 1) — n{i)\ < 1 there exists a point z G Z so that 
F'{z) G for all i > 0. 



One obvious consequence of the random walking described in the theorem, is the following 
unusual behavior. Take e > small and define the local and unstable stable set corresponding 
to rate r as 

W/'^(r) := {x; dist(0t(x), r) < e for alH > and lim ^ log(dist(0f (x), T)) ^ r} 
W"'^(r) := {a;;dist(0t(x),r) < e for alH < and lim ^ log(dist(0f (x), T)) ^ r}. 

t^-oo \t\ 

Then the above system has for each e > and each t > close to zero, that both W^''^ and 
W^''^ are non-empty in any neighbourhood of T. 

Orbits spiral around T with a tighter pitch, the closer the orbit is to F, see Fig[T] This is 
the reason why the first return map to a section Z can have an infinite number of fixed points, 
accumulating to Z (IT. The reason why one has such strange dynamics, is that the first return 
map P: Z ^ Z near to T has a very special form. If we identify Z with and T D Z with 
G M^, then P has essentially a composition of maps of the form 

F{x) = i?i/||^||(a;). 
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Here ||(xi, a;2)|| — |xi| + |x2| is the Zi norm on M^, i?t is a rotation over angle < leaving the 'circles' 

/ 2 

in the h norm invariant (i.e. ||i?t(x)|| = ||a;||) and A is a matrix of the form A 



0.5 

Note that is a homeomorphism, which is smooth. (The actual first return map is piecewise 
affine, which is one reason why the actual return map has a more complicated expression.) The 
image of a ray through is under the map -F is a spiral, see Figure |2] There exist r G (0, 1) so that 
F can send a point in the annulus Fn := {x G M^; r*"*"^ < < r*} to annuli F^-n, ■ ■ ■ , Fn+N 
in a way which almost completely 'forgets' where the point started from. So the dynamics can 
be essentially modelled by that of a random walk. Indeed, it was proved in |2009j that there are 
orbits which can go to at a prescribed speed. The image of a set {x gM.'^;\\x\ \ = 1} under the 
first six iterates of F is drawn in Figure [3] 



D 
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Figure 1: On the left, the haxagon F. A disc D with boundary F and a section Z are also drawn. Both 
D and Z are sections. There exists a sequence of fixed points (and points of higher periods) accumulating 
to 2 = Z n F of the first return map to Z. In the middle two figures, orbits are drawn near a piece of 
r. Orbits spiral along rectangular tubes around F. Orbits starting nearer to F spiral with a finer pitch 
around F, see the 2nd figure on the middle. Focally orbits spiral on these rectangular tubes, but it is 
certainly not true that orbits remain on topological tori around F, because the tube is skewed when it 
has fully followed the entire orbit F, see the figure on the right. 



I 




Figure 2: On the left, the image of a curve / through is a spiral. (Under the model map F defined 
above, the image of I looks slightly different.) A conveniently chosen curve I contains a sequence of fixed 
points, converging to 0. On the right, a sequence of annuli is drawn. There exist orbits which visit annuli 
An(i) where n{i) can be chosen arbitrarily so that \n{i + 1) — n{i)\ < 1 and n{i) > 0. 



Another way of visualising the dynamics is by considering a global first return section. 

Theorem 1.3 (Existence of a global first return section through F). For H as above, there exists 
a topological disc D C H^^{1) whose boundary is equal to F so that the following properties hold: 
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Figure 3: The image of under the first six iterates under x i-^ Ao i?]^/||^||(a;) of the set = 1. 

• The orbit through each point x e \ F intersects the topological disc D infinitely 
many times (and each of these intersections is transversal). So D is a global section with 
a well-defined first return map. 

• The first return time of z to D tends to zero and the first map tends to the identity, when 
ze D\T tends to z' e F. 

That it is possible to find a disc D whose boundary is equal to an orbit F, which is a global 
section and so that it extends to the identity on the boundary, is related to the fact that the 
flow coils around F as in Figure [l] This is of course rather different from what happens when a 
vector field is smooth, and so such a disc D certainly could not exist if the vector field had been 
smooth. Figure |4] shows an orbit of the first return map to D. 
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Figure 4: A projection of the topological disc D and the orbit of a typical starting point under 
the first return map to D. There are two 'egg-shaped' regions which are permuted by the first 
return map. Orbits within this region form invariant circles (these 'egg-shaped' regions have 
period two) . At the center of these regions there is a periodic orbit of period two. Locally, the 
map is a rigid rotation over an angle which is determined by the parameter /3. For 13 = a the 
rotation angle is a irrational number, so orbits under the first return map lie on circles. 



1.2 Relationship with zero sum games (and some new results about 
fictitious play) 

The final two theorems of this paper show that the dynamics that we introduced in the previous 
two theorems, actually are naturally associated to zero-sum games. In this way we will also shed 
new light on dynamics associated to zero-sum games. 

Brown |1949j suggested a dynamical system which can be used to approximate the Nash 
equilibria of a zero-sum game and it was shown Robinson |1951j that indeed solutions converge 
to the Nash equilibria of this game. In the continuous case this system is given by the differential 
equation 

^ BIlAip^) - p\ ^ = BRsip^) - . (1.5) 

Here 

BJZa{p^) ■■= argmaxp^ A^p^ and BRb{p^) arg min . (1.6) 
Some authors prefer a different time parametrization (taking s — e*), which gives 

^^-{BIZMp^)~P% ^ = -{B}Zn{p^)^p% (1.7) 
as s as s 

Here p^ e T,a C M™,p^ e Ss C M" be the space of probability vectors in E™ resp M". Here 
p^,p^ represent the strategy profile of player A resp. B, see Section |2] and BRa and WZb are 
defined similarly as before, see |2.10| and |2.11[ Note that Ai is now a. m x n matrix with m 



and n not necessarily equal. The latter parametrisation allows for a learning interpretation of 



the dynamics, see |1998j . Sometimes the differential equation (1.5 1 is called the best-response 



dynamics and (1.7 1 fictitious play dynamics. 

Even though orbits converge to the Nash equilibrium of a game, one can ask how this con- 
vergence goes. We will describe this first in the case when n = m and the Nash equilibrium is 
unique and completely mixed. In this case write E = = E^ and let p = p"^ and q — p^ . 
To describe how convergence occurs, we decomposes the differential equation in a radial and a 
'spherical' direction (in a similar way to writing a vector field in polar or spherical coordinates). 
This can be done as follows. Let as before 

H{p'^,p^)^ max p'^Mp^ - min p"^ Mp^ ^ BJZa{p^) Mp^ - p"^ MBJIb{p^)- 

Note that H >0 since ER,a{p^)Mp^ > p'^Mp^ > p^MB1Zb{p^) and note that H{p,q) = 
since {p, q) is a completely mixed Nash equilibrium. Take a half-ray 1+ through p, q. The function 
H is monotonically increasing on In particular, Ij^ intersects H~^{1) in a unique point and 
so we can define 

7r:ExE\{p,g}->ff-i(l) 
to be the corresponding map. In this way, we have defined 'spherical coordinates': 

E X E \ {p, g} 9 (p, q) ^ (r, e M+ X H-\l) 

where 

r = H{p, q) and (j) ~ ■n{p, q). 
Theorem 1.4. Assume that n = m and the two-player zero-sum game has a unique completely 



mixed Nash equilibrium. Then, in these spherical coordinates, best response dynamics (1.5) 
becomes 



and fictitious play dynamics {1.1) becomes 



where X is piecewise constant (and so the flow is a piecewise translation flow). Moroever, X is 
linearly equivalent to the Hamiltonian vector field on H^^{1) « 5^"^'^ corresponding to 



dp _dH dq _ dH 
dt dq ' dt dp 



(which has no stationary points). 



So orbits converge to the Nash equilibrium with r(t) — ce~* (resp. r(t) = c/t) and with 
the motion of (f> similar to that of a Hamiltonian flow. Note that Hamiltonian flows are volume 
preserving and therefore do not have attracting periodic orbits. In the trivial case of a 2 x 2 game, 
then H^^{c), c > 0, is the boundary of a quadrilateral and the Hamiltonian flow moves either 
clockwise or counter clockwise, see Figure |5] But in the higher dimensional case of a fc x fc game 
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with fc > 3, typical orbits will have chaotic motion in the spherical direction since a Hamiltonian 
flow preserves volume. For typical starting points, players switch strategies erratically. 

One application of related to the main theorem in Krishna and Sjostrom |1998j . This theorem 
states that for generic games, it is impossible for an open set of initial conditions to all converge 
to the Nash equilibrium with the same cyclic play (i.e. choosing the same periodic sequence of 
strategies). The corollary shows that this theorem does NOT hold in general: 

Corollary 1.1. For zero-sum games it is possible that an open set of initial conditions converges 
with cyclic play to the Nash equilibrium. 



Proof. Consider the game (1.4 1 from Theorem 1.2 This game has an orbit which converges 



periodically to the Nash equilibrium, see |2009j . This orbit corresponds to a periodic orbit for 
the induced flow on the level set. If one considers the Poincare flrst return map to the global 
section from Theorem |1.2| then this map has a periodic point of period 2 at the intersection of 
this orbit with the section (these two points are at the 'centre' of the two egg-shaped regions 
in Figure |4| . From [2009^ and |2008j it follows that the multiplier of this first return map is on 
the unit circle. Since the first return map is locally a rotation, it follows that orbits stay on two 
circles around this periodic two orbit. When the parameter /3 is chosen to be equal to the golden 
mean a = {Vb— l)/2, then the rotation has irrational angle, and the orbits spiral dense on these 
circles. For fictitious play, this gives that orbits spiral towards the Nash equilibrium along the 
cones over these circles with apex the Nash equilibrium. □ 



Since the differential equations (1.5 1 and (1.7 1 are differential inclusions and not differential 



equations, and moreover the right hand side of these are not Lipschitz, one cannot guarantee 
uniqueness of solutions and continuous dependence on initial conditions. However, provided 
the transversality conditions from the introductions hold, all these difficulties are absent. More 
precisely, on the way to proving Theorem |1.1[ we also prove the following theorem (where the 
new part is the last statement, continuity of Fictitious play): 

Theorem 1.5. Assume that we have a m x n two-player zero-sum game with m and n not 
necessarily equal. Assume that holds. Then 



the Nash equilibrium {p^,pf) is unique; 



• there exist G {^t ■ ■ ^ 'ti} and ji, . . . , jV G {^t ■ ■ ,n} so that p^ G (ej^ , . . . , 6^^,) C 

°ji I ■ ■ ■ ' 



J^A o-nd p^ G (ejj , . . . , Bj^) C and so that moreover the ii, . . . ,ir-th component of p^ 



and ji, . . . ,jr-th component of pf are non-zero (so {p^,pf) is a completely mixed Nash 
equilibrium w.r.t. to this subgame); 

• the other components of p^,p^ decrease monotonically with time for any solution of the 



differential equations (1.5) and {1.7) 



(1.5) and (1.7) define continuous flows. 



Here we write (e^ ^ , e^^ , . ■ . , e^j ) for the part of the boundary of where only the ii , i2 , . . . , «z 
coordinates are non-zero ((e^j, . . . , Cj^) C S_b is defined similarly). This theorem tells us that the 
dynamics decomposes into a monotone movement towards (e^j , . . . , e^^) x (e^j^ , . . . , e^^) C Sa x 
and a movement inside the r x r subgame (ei^ , . . . , e^^) x {cj^ , . . . ,ej^). 
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Figure 5: A 2 x 2 zero-sum game converges to the Nash equihbrium. The level set H ^(c) is a 
quadrilateral and the Hamiltonian flow moves periodically on this. 



1.3 The organisation of this paper 

In Sections [2] we shall prove the uniqueness of the Nash equilibrium and the first three parts of 
Theorem |1.5t Then in Section |3] to [5] we shall prove Theorem 1.4 Theorem 1.2 is proved in 



van Strien & Sparrow |2009j . In fact, in that paper we study a family of systems which includes 
non-zero sum games and show that the random- walk behavior from Theorem |1.2| occurs in some 
induced dynamics. In the present paper we show that the zero-sum case corresponds Hamiltonian 
dynamics, so the proof of Theorem |1.2| is a combination of the results from van Strien & Sparrow 
^2009j and Theorem 1.4 Finally we shall prove Theorem |1.1| and Theorem 1.5 in Section |6] 



2 Zero sum games 

In this section we will show that in the present setting, the Nash equilibrium is unique if M is 



invertible or if the trans versality assumption (1.2 1 holds. This will in particular prove the first 
three part of Theorem |1.5| 



Consider a two-player zero-sum game with n strategies. This means that players A and B 
each choose a probability vector p"^ and in K™ resp. E". For convenience, let C M™ and 
Tib C K" be the space of probability vectors in M™ resp. M". So p^ £ describes the strategy 
profile of player A and p^ &T,b that of player B. Traditionally, one lets p^ be a row vector and 
p^ a column vector. (In the Hamiltonian context we assume that n — m and let p — (p^)' and 
q = p^ ■) The utility player A gets when p^ and p^ are played is determined by some function 
M. {p^,p^). Usually this function is of the form p"* M.p^ where M. is a m x n matrix. Since the 
game is zero sum, player B gets utility —p'^M.p^. So player A is trying to maximize her utility 
p"^ Mp^ , i.e., for each choice of strategy profile p^ of B, player A chooses a strategy profile 

e EJlAip^) ■■= argmaxp^TWp-^. (2.10) 

pA 

On the other hand, B tries to maximize the negative of this expression, i.e., tries to minimize 
p"^ M p^ and so will play 

p^ e Eflsip^) := argminp^TWp^. (2.11) 

A Nash equilibrium is a choice of strategies from which no unilateral deviation by an individual 
player is profitable for that player. That is, {pt,pf) is a Nash equilibrium if 

pt e BJZAipf) and pf e BlZeipt)- (2.12) 
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Equivalcntly, {pf,pf) is a Nash equilibrium if 

M{p^ , ) = maXpA miUpB p^ M p^ = miUpB maXpA p"^ M p^ , 

p^ e argmaXpA miupij M {p^,p^) (2-13) 

pf G argmiUpB maXpA M {p^,p^). 

Here A4 {pf ,pf) := p^Adpf is called the value of the game. A similar definition can be 
given for many player games, but in the situation we consider of two-player zero-sum game the 



notion (2.121 of Nash equilibrium coincides with that of a minimax equilibrium (2.131 (a notion 
introduced earlier by von Neumann for two-player games). 

It was shown by von Neumann that each zero-sum has a unique value, and that each zero-sum 
game has at least one Nash equilibrium (or minimax solutions as von Neumann called them). 
Moreover, the set of Nash equilibria is convex. However, in general the Nash equilibrium is not 
unique, see for example Section [7] In this note we primarily consider the situation that there 
exists a Nash equilibrium {p^ ,pf) which are completely mixed (so all components of these vectors 
are strictly positive). 

Lemma 2.1. Assume that m = n, that M. is an invertible matrix and that {pf ,p^) is completely 
mixed Nash equilibrium of the game. Then {p^,p^) is the only Nash equilibrium of the game. 

Proof. Since is a best response to and all components of pf are strictly positive, all 
components of pf M are equal. Hence pf is a multiple of (11 ... 1) M~^. Since pf is a 
probability vector, pf is uniquely determined (among all completely mixed Nash equilibria). 
Similarly, all components of A4pf are equal, and again p^ is uniquely determined (among 
all completely mixed Nash equilibria). Note that if not all coordinates of A4p^ are equal 
then BR!^{p^)Mp^ > p^Mp^ = p.,Mpf (here > holds because then BJ^{p^) puts no 
weight on some of the coordinates). Similarly, if not all coordinates of p^ A4 are equal then 
P^MBRP{p^) < p"^ Mp^ = pf Mpf . It follows that whenever {p^,p^) ^ {pt,P*) then 
{p^,p^) does not have the same value as the Nash equilibrium, and so is not a Nash equilib- 
rium. □ 

The proof shows that it is enough that the sets {p^ e M™; ^ v} and {p^ Mp^ ^ 

v'} are one-dimensional, where ?; = (1 1 . . . 1) and v' is its transpose. This condition is slightly 
more natural than requiring that M is invertible. Indeed, the matrices A4 and M are equivalent 
as far as the games are concerned when Ai = aA4 + b with a,b E R. But within the class of 
equivalent matrices, the property of invertibility is not preserved, whereas the latter condition is. 

Since we also want to consider two-player m x n zero-sum games, let us prove the analogous 
result for such games. 



Lemma 2.2. Assume that (1.2) holds. Then we have the following properties. 



• Consider a linear space Za C C M™ where player B is indifferent between strategies 
ji, j2, . . . , jfc G {l,...,n}. Moreover consider a r- dimensional face (e^j^ , . . . , Ci,,) of Y^a. 
Then Za H (e^^ , . . . , 6^,,) has codimension {k — 1) as a subset of (e^^ , . . . , e^^) (or is empty). 
A similar statement holds when the roles of players A and B are interchanged. 
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• The Nash equilibrium {pf,p^) of the game is unique. 

• there exists ii, . . . , v G {1, ■ • • , ™} cmd ji, . . . ,jr € {1, ■ . ■ ,n} so that p^ € (e^j , . . . , e^^) C 

T,A and pf € (gj-j , . . . , Cj,.) C Ss anrf so that moreover the ii, 12, . . . , V components of p^ 
and the ji, j2, ■ ■ ■ ,jr coordinates of p^ are non-zero (so {pf^,pf) is a completely mixed Nash 
equilibrium w.r.t. to this r x r subgame). At p"^ player B is indifferent between strategies 
ji, . . . ,jr and all other strategies are less preferred. Similarly at pf player A is indifferent 
between strategies zi, . . . , v and all other strategies are less preferred. 

A more abstract way of rephrasing the first part of this lemma is that all the boundary faces 
of Ea (of any dimension) are in general position w.r.t. the linear spaces where player B is 
indifferent to a number of strategies. (Analogously to the fact that two lines in M.^ typically will 
miss each other.) 

Proof. Consider a linear space Za C Ea C M™ where player B is indifferent between strategies 
jiij2, ■ ■ ■ ,jk G {!,..., n}. This space is equal to the set of p^ G where p"^ M' ~ 0, where 
M ' is the (fc — 1) X n matrix made up from the difference of the j2, . . . , jfe-th and the ji-th 
column vector of the matrix M. . By only considering the matrix M. * which consists of only the 
ii, . . . ; v-th rows of M. we get the intersection of Za with (e^j , . . . , e^^). Since, by assymption 



( 1.2 1 the matrix M. ' has maximal rank, it follows that this intersection has codimension (A: — 1) 
in (cij, . . . , Bi^). (So if r < A: — 1 then the intersection is empty. The intersection will be also 
empty when the intersection of Za with the linear space through (e^^ , . . . , e^^) is outside S^-) 

Let {p^,pf) be a Nash equilibrium and assume that player B is indifferent between k (optimal) 
strategies and say player A is indifferent between I (optimal) strategies and to be definite assume 
that k < I. Let ji,. . . ,jk G {1, . . . ,n} be these optimal strategies for player B. Note that by 
the first part of the lemma we have that k < m. Since only the ji, . . . ,jk strategies are optimal 
for B, we get that p^ G i^jn ■ ■ ■ j^jk) ^ ^b- But this implies (by the first part of the lemma) 
that player A can only be indifferent between at most k strategies (and between at most fc — 1 
strategies if pf is also contained in some lower-dimensional sub-simplex of {cj^ , . . . , ej^)). Since 
we assumed that k < I we get that k = I and that all these strategies are optimal (and that the 
ji, . . . , jk-th components of are strictly positive). If ii, . . . ,ik G {l,...,m} are the optimal 
strategies of A, then G (e^j , . . . , 6^^.) C (and the ii, . . . , i^-th components of are strictly 
positive. 



Using Lemma 2.1 the Nash equilibrium is unique within this fc x fc subgame. If the game has 
two Nash equilibria, then a convex combination of these is also a Nash equilibrium and so there 
would be a completely mixed Nash equilibrium in a fc' x fc' subgame with fc' > fc (containing 
the fc X fc subgame from before) . But this contradicts the uniqueness of completely mixed Nash 
equilibrium we already established. □ 



Fictitious play and speed of convergence to the Nash 
equiUbrium 



In this section we will prove that the 'r-part' of (1.8) and (1.9l in Theorem 1.4 is a claimed 



Brown |1949j suggested the dynamical system which can be used to approximate the Nash 
equilibria of a game and it was shown Robinson |1951j that indeed solutions converge to the Nash 
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equilibria of this game. In the continuous case this system is given by the differential equation 
Some authors prefer a different time parametrization (taking s — e*), which gives 



(3.14) 



ds 



1 



dpB 
ds 



1 



(67^B(p^)-p^). 



(3.15) 



Strictly speaking, the equations (3.14) and (3.151 are differential inclusions, but it is not hard to 
show that it has solutions nevertheless, see |1984j '). One can also consider a discrete version of 
these dynamical systems, but we will not consider this here. 

Even in the case when the players are involved in a general sum game, the above differential 
equations make sense (in this case BJZa and BJZb are defined similar as before by I3JZa{p^) ■= 
a,r g max p'^Ap^ and !3JZb{p^) := aigmaxp^Bp^ where A and B are two different matrices ^4). 
There are many papers which show that one has convergence to the equilibrium in particular 
situations: for games where one or both of the players have only 2 strategies to choose from, see 
Miyasawa [TMT] and Metrick & Polak P^M] for the 2 x 2 case; Sela [21100] for the 2 x 3 case; 
and Berger [2005] for the general 2 x n case. Jordan |1993j constructed a 2 x 2 x 2 fictitious 
game with a stable limit cycle. The 3x3 example studied in this paper, shows that the situation 
is far more complicated in general, see Sparrow at al |2008j and van Strien & Sparrow |2009| . 
Theorem |1.2| summarises some of the results from these last two papers. 

Another important reason for studying these differential equations is because they can be 
used as a model for rational learning, see for example Fudenberg and Levine |1998j . 

It is easy to prove Robinson's result (in the continuous case) and to show the r part of 'r-part' 
of (1.8 1 and (1.9 1 is as claimed in Theorem 



1.4 



Lemma 3.1. 



Hit) 



H and (1.1) gives 



Consider H{p^ . 
dH 



p^) = BR.a{p^)Mp^ -p'^MBRsip'^)- Then |i.5p gives 



dH 
~dt 



—H/t. So H tends to zero along orbits (as H(t) — ce * resp. 
c/t where c depends on the initial condition). Furthermore, the zero set of H coincides 



with the set of Nash equilibria of the game, 
is continuous at this Nash equilibrium. 



Moreover, if the Nash equilibrium is unique, the flow 



Proof. To show that solutions of ( 1 . 5 ) and ( 1 . 7 1 tend to Nash equilibria, we show that H {p ,p ) = 
BIZa [p^ )Mp^~p^M BJZb (p'^ ) is a Lyapunov function. Note that BJZa {p^ )Mp^ >p^Mp^ > 
p"^ M BJZsip"^). That is, i? > and i/(p^,p^) = iff (p^,p^) is a Nash equilibrium. Since BJZa 



and !3JZb are piecewise constant, (1.5 1 implies 

dpB 



dH 
~dt 



BRaIp^M 



dt 



dp^ 
~dt 



MBRBipn 



BJZAip"") M {BJZb{p^) - p^) - (67^A(P^) - P^) M BR.b{p^) = -H. 



(Similarly, 



implies 



dH 
~dt 



-H/t). 



□ 



This use of as a Lyapunov function goes back to Brown [1949 and was explicitly mentioned 
in Hofbauer |1995j . Harris |1998j used a related method to analyse the speed of convergence to 
the value of the game. 
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4 The induced flow associated to fictitious play is piecewise 
a translation 



In this section we will compute the induced differential equation which we obtain after projecting 
(1.51 and (1.7 1 onto a level set of H. In this section we shall not assume that n = m. 

The function H{p^,p^) — BIZa{p^) Mp^ — p^ MER-Bip^) is continuous, because of the 
definition of BIZa and EJZb- It is even piecewise affine (it is locally affine on sets where EJZa and 
BIZb are locally constant). Let us denote the Nash equilibrium of the game by {E^,E^). On each 
half-ray starting at {E^, E^) the function H is strictly increasing and, moreover, H{E^, E^) = 0. 
It follows that for each c > 0, H^^{c) is topologically a sphere, i.e. homeomorphic to S'^"^^. 
The orbit of the first differential equation is 

p^{t) = e- V + (1 - e-')BJZA{p'') ,p''{t) = + (1 - e-*)ef 

Let us project this from {E^,E^) onto a level set of H. That is, take 

p^{t) - (1 - s{t))p^{t) + s{t)E^,p^{t) = (1 - s{t))p^{t) + s{t)E^ 

with s{t) so that the curve remains within a level set of V. To do this, we have to choose s{t) so 
that 

M [(1 - s{t)) {e-^pB + (1 - e-*)67^B(p^)) + s{t)EB] - 

[(1 - sit)) (e- V + (1 - e-*)67^A(p^) ) + s{t)E^] M BJZb{p^) = 

= EIZa{p^) Mp^ -p^MB}ZB{p^)■ 
l<lote that E^MBIIb{p^) = E'^ME^ = BIZa{p^) M E^ and therefore the previous equation 
is equivalent to 

{l-s{t))e-' [BJZa {p^ ) Mp^ ~p'^MBR.b{p^)]=BR.a {p^ ) Mp^ - p"^ MBR.b{p^), 
i.e., to 1 — s{t) = e* and s{t) = 1 — e*. So the orbit becomes 

p^{t) = e*p^{t) + (1 - e*)£;^ = e*e- V + (e* - l)B7^A(p^) + (1 - e*)E^ 
pB^t) ^pB + (e* - l)mBiP^) - E^). 



So 



dp"^ 



e\mA{p'') -E' 



dp^ 
dt 



e*iB}ZB{p^)-E^). 



If instead of (1.5l one uses (1.7l then one gets (using that s = e*), 
dp"^ 



= BJZa{p^)~-E^, 



BRb{p'') - E' 



ds ds 

This is obviously a translation flow, because the right hand side is piecewise constant. This 
completes the proof of equations (1.8 1 and (1.9). Note that the induced flow on level sets of H 
is volume preserving system. 
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5 The induced flow is Hamiltonian 

Let assume now that n = m and consider again 

To make the role of p^,p^ symmetric, let us write p^ = Tr{p^), = Tr{E^), BR,a{p^) = 
Tr{BR,A{p^)) and rewrite H as 

H{p^,p'') = Tr{M)BJZA?') ■ p"" - MEJlBip^) ■ p^ , 

where • stands for the inner product. Notice that € Sa, € Ss- The derivatives restricted 
to TiA and S b are of the form 

/ 1 \ /I 
^ = MBRsip^) -Xa\ : I ««id ^ = Tr{M)B}ZA{p'') - Xb 

where Xa,Xb are chosen so that the vector lies in the tangent space of S^, S^- In other words, we 
need to ensure that the sum of the vectors add up to zero. Notice that A4 is also of the form 
/ 1 



A'. 



3J-T f)M 
^ so we can write = - M {BJIb{p^) - nE^) and ^ = Tr{M){BllA{p^) - n'E^) 

for sonic /i,///. However, E^ MEIZBip"^) = E^ M E^ , so one needs to take //, = 1 to get that 
the sum of the components of the vector add up to zero. Similarly jj' = 1. It follows that 



^ = -M{BRb{p^)-E'') and ^ = Tr{M){mA{p^) - E^) . 
It follows that the differential equation from the previous section can be written as 



(^,^) =iBlZA{p^)-E^),BRBip^)-E^) 



= (Tr{M)-'^,-M-'—\ 



dp^ J 

( ^ \ 



(rr(A^))-i 




A 



dp 

dH 
\dpB J 



This is linearly equivalent to a Hamiltonian system in standard form. Indeed, take matrices A 
and B, and take x = Ap^ and y = Bp^. Then 
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dy dp^ dH , ,,dH 

Taking B — Id and A — Tr{M)^^ (or B — and A — Id) gives then the usual Hamiltonian 

form 

, _ m 

dy 



y 



dH 
dx ' 



6 Continuity of zero-sum fictitious play 

In this section we shall show that fictitious play defines a continuous flow, provided the transver- 
sality condition from the introduction of the paper hold. In this section we shall not assume 
that m = n. 

Along the set S where the player A is indifferent between stategies i and j, BR(p^) is 
not a singleton and so the flow is not uniquely defined, and we will not be able to get continuity 
in general. In this section we will deal with this issue. That one needs to make assumptions 
follows from the example described in Figure |6] 




Figure 6: The case wlien M is the 3x3 identity matrix. The flow is multi-vaiued and discontinuous 
at any several codimension-one sets, for example when (p^ ,p^) £ Ea x Es is a point as marked by *. 



To illustrate the proof that the flow is continuous, let us refer to the 2x2 zero-sum game, 
see Figure |5] At the Nash equilibrium, the vector field is multivalued (and each directions is 
allowed). However, an orbit cannot move out of the Nash equilibrium, since the orbit through 
each point which is close to the Nash equilibrium remains close to it. A similar situation also 
occur in the higher dimensional case, see Figure [7] In that figure, the orbit is multivalued along 
some dashed line. However, nearby orbits spiral along a rectangular cone (in a manner which is 
similar to the 2x2 zero-sum game). It follows that the resulting flow becomes continuous. 

Theorem 6.1 (Transyersality condition implies continuity and uniqueness). The motion de- 
fined by equations (1.5) forms a continuous flow outside {E^,E^) provided the transversality 
assumption (1.2) from the introduction holds. 



Proof. Let us first show that the above condition is enough to get that orbits only momentarily 
stay in the codimension-one planes in which (only) one of the players is indifferent. The linear 
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space of vectors in where the I and I'-th strategy are indifferent to player B consists of the vectors 
in T,A C M™ which are orthogonal to the difference of the l-th and ^'-th column vector of the 
matrix A4 . Therefore if all cocfhcient within each row vector of A4 arc distinct then, for any 
k, the fc-th unit unit vector in M™ is not contained in this indifference set. Hence, an orbit 
immediately moves away from the codimension-one planes where only player B is indifferent 



(i.e. it moves transversally across these planes). (Note that (1.2) requires in particular that all 
coefficients of the matrices A4 ' are non-zero, and hence all coefficient within each row vector 
of M. are distinct.) The analogous statement holds for the codimension-one planes where only 
player A is indifferent. 

Let us call the part of where precisely r components are non-zero, a r -dimensional face of 
TiA- We shall write {ci-^ , e^^, . . . , e^j) for the part of the boundary of T^a where only the ii, i2, . . . , i/ 
coordinates are non-zero. 

Let us next consider the situation that the players are both indifferent between several strate- 



gies. In this case the issue is that the right hand side of (1.51 is not single valued. (So we 
have a differential inclusion rather than a proper differential equation.) To analyse this situa- 
tion, consider a linear space Za C T,a C where player B is indifferent between strategies 
ji, j2, ■ ■ ■ , Jfc € {1, . . . , n} and a linear space Zb C Sb C where player A is indifferent be- 
tween strategies ii,i2, ■ ■ ■ ,ii G {1, . . . , m}, and consider a point p^,p^ where these strategies are 
optimal. The number of directions in which player A can move (starting from p'^jP^ towards 
Cij, . . . , Ci, while staying in Za is equal to the dimension of Za H (e^j, e^^, . . . , 6^,). By the first 



assertion of Lemma 2.2 the dimension of Za H (e^j , e^^ , • ■ • , e.;, ) is equal to {I — I) ~~ {k — 1) = I — k 
(in fact, even if I > k the intersection can be empty because (e^j , . . . , e^,) is not some linear 
space, but only a simplex within such a linear space) . Similarly interchanging the role of A and 
B, the number of directions player B can move while staying in Zb has at most dimension k — I. 
It follows that only when k — I the players two players can move while staying within Za and 
Zb, and then there is only one direction in which they can move. So there is at most one orbit 
starting from p'^^p^ which stays within Za, Zb- 

Note that even if there exists an orbit which stays within Za and Zb there may be another 
solution in which moves off this set. What we will show an orbit which starts slightly off the set 
Za, Zb, orbits spirals around it. (Here we use that the game is zero-sum.) In this way, it follows 
that the flow becomes continuous and unique. 

To prove this, let us use the notation from before and notice that we have shown that only 
when k = I the orbit does not necessarily move transversally through Za x Zb ■ So locally near a 
point of Za x Zb only strategies ji, . . . ,jk are optimal for player B and strategies ii, . . . ,ik are 
optimal for player A. But one can decompose the flow as a flow where one moves towards the 
plane (e^^ , . . . , e^j.) C and (cj^ , . . . ,ej^) C S^, and a flow which is parallel to {e.i^ , . . . , e^j.) x 
(e^j , . . . , ) . So the latter can be described as fictitious play associated to the corresponding 
k X k subgame with Nash equilibrium 

{ZaH {e,,,...,e,,)) X (^3(6^-^ , . . . , Cj,)) . 

By Lemma [3.1 1 the latter flow is continuous. □ 



Note that everything in the proof of Theorem 6.1 except the last part, has an obvious analogue 
for non-zero games. 
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Figure 7: This picture illustrates what happens when both players are indifferent between two strategies. 
The dashed line denotes this set. The targets are drawn marked as •, and by choosing appropriate 
combinations of the targets the players can move along the dashed line. Nearby orbits spiral along some 
cone towards the apex of the cone, always aiming for one of the points marked as •. Note that the vector 
field on the cone is not close to the vector field along the dashed line. The reason orbits move like this 
is that the 2x2 game which describes how both players alternate between the strategies is zero-sum. 
Note that orbits of a zero-sum 2x2 game always orbits spiral, and cannot have a saddle-point. 



7 Example of a zero-sum game with a non-unique Nash 
equiUbrium on the boundary (for which fictitious play is 
non- cont inuous ) 



Consider the zero-sum game determined by A4 
minimizes w.r.t. A4 . Since 



1 
1 2 



Pb + 2p| 



and p^M = (-1 



(0 1) when pf < I 
Y^A when pf — 1, 



Player A maximizes and B 

when p^ e [0,1/2), 
when pi — 1/2, 
when p^ e (1/2,1]. 



This shows that the Nash equilbria consists of the segment 

{(p-^,P^)eS^xEs;pfe [0,1/2], p^ 



} 



and that fictitious play moves towards the line segment consisting of Nash equilibria. However, 
an initial condition is chosen with pf < 1 will end up in the right end-point of this segment (i.e. 
p^ = and pf = 1). Note that the flow is not continuous in this case, see Figure [s] 
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